[CI] Enable "Rerun Failed Tests" for Small Tests #4586

tomaszwojcikowski · 2025-12-02T04:51:10Z

This PR enables CircleCI's "Rerun failed tests" for small tests.

codecov · 2025-12-02T05:01:01Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.00%. Comparing base (8022a7d) to head (30f28e5).
⚠️ Report is 15 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4586      +/-   ##
==========================================
+ Coverage   85.95%   86.00%   +0.04%     
==========================================
  Files         565      565              
  Lines       33930    33930              
==========================================
+ Hits        29166    29181      +15     
+ Misses       4764     4749      -15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…o include extra args for test suite

mongoose-im · 2025-12-03T11:35:13Z

dynamic_domains_mssql_mnesia_28 / odbc_mssql_mnesia / a560aaf
Reports root

elasticsearch_and_cassandra_28 / elasticsearch_and_cassandra_mnesia / a560aaf
Reports root/ big
OK: 683 / Failed: 0 / User-skipped: 72 / Auto-skipped: 0

small_tests_28 / small_tests / a560aaf
Reports root / small

small_tests_28_arm64 / small_tests / a560aaf
Reports root / small

small_tests_27 / small_tests / a560aaf
Reports root / small

ldap_mnesia_28 / ldap_mnesia / a560aaf
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1359 / Auto-skipped: 0

dynamic_domains_mysql_redis_28 / mysql_redis / a560aaf
Reports root/ big
OK: 5186 / Failed: 0 / User-skipped: 157 / Auto-skipped: 0

ldap_mnesia_27 / ldap_mnesia / a560aaf
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1359 / Auto-skipped: 0

internal_mnesia_28 / internal_mnesia / a560aaf
Reports root/ big
OK: 2499 / Failed: 0 / User-skipped: 1218 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_28 / pgsql_mnesia / a560aaf
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

pgsql_cets_28 / pgsql_cets / a560aaf
Reports root/ big
OK: 5311 / Failed: 0 / User-skipped: 195 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / a560aaf
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

mysql_redis_28 / mysql_redis / a560aaf
Reports root/ big
OK: 5600 / Failed: 0 / User-skipped: 149 / Auto-skipped: 0

cockroachdb_cets_28 / cockroachdb_cets / a560aaf
Reports root/ big
OK: 5311 / Failed: 0 / User-skipped: 195 / Auto-skipped: 0

pgsql_mnesia_28 / pgsql_mnesia / a560aaf
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0

mssql_mnesia_28 / odbc_mssql_mnesia / a560aaf
Reports root/ big
OK: 5609 / Failed: 0 / User-skipped: 140 / Auto-skipped: 0

pgsql_mnesia_27 / pgsql_mnesia / a560aaf
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_28 / odbc_mssql_mnesia / a560aaf
Reports root/ big
OK: 5216 / Failed: 0 / User-skipped: 127 / Auto-skipped: 0

mongoose-im · 2025-12-03T13:07:48Z

elasticsearch_and_cassandra_28 / elasticsearch_and_cassandra_mnesia / 0638a02
Reports root/ big
OK: 683 / Failed: 0 / User-skipped: 72 / Auto-skipped: 0

small_tests_27 / small_tests / 0638a02
Reports root / small

small_tests_28 / small_tests / 0638a02
Reports root / small

small_tests_28_arm64 / small_tests / 0638a02
Reports root / small

ldap_mnesia_28 / ldap_mnesia / 0638a02
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1359 / Auto-skipped: 0

ldap_mnesia_27 / ldap_mnesia / 0638a02
Reports root/ big
OK: 2357 / Failed: 1 / User-skipped: 1359 / Auto-skipped: 0

graphql_server_SUITE:admin_cli:clustering_tests:join_successful

{error,{#{what => invalid_response_code,
      response_code => {exit_status,1},
      expected_type => ok},
    [{graphql_helper,assert_response_code,2,
             [{file,"/home/circleci/project/big_tests/tests/graphql_helper.erl"},
              {line,258}]},
     {graphql_helper,get_ok_value,2,
             [{file,"/home/circleci/project/big_tests/tests/graphql_helper.erl"},
              {line,241}]},
     {graphql_server_SUITE,join_successful,1,
                 [{file,"/home/circleci/project/big_tests/tests/graphql_server_SUITE.erl"},
                {line,153}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1303}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1235}]}]}}

Report log

dynamic_domains_mssql_mnesia_28 / odbc_mssql_mnesia / 0638a02
Reports root/ big
OK: 5216 / Failed: 0 / User-skipped: 127 / Auto-skipped: 0

dynamic_domains_mysql_redis_28 / mysql_redis / 0638a02
Reports root/ big
OK: 5186 / Failed: 0 / User-skipped: 157 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_28 / pgsql_mnesia / 0638a02
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 0638a02
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

internal_mnesia_28 / internal_mnesia / 0638a02
Reports root/ big
OK: 2499 / Failed: 0 / User-skipped: 1218 / Auto-skipped: 0

pgsql_cets_28 / pgsql_cets / 0638a02
Reports root/ big
OK: 5311 / Failed: 0 / User-skipped: 195 / Auto-skipped: 0

pgsql_mnesia_27 / pgsql_mnesia / 0638a02
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0

mysql_redis_28 / mysql_redis / 0638a02
Reports root/ big
OK: 5600 / Failed: 0 / User-skipped: 149 / Auto-skipped: 0

cockroachdb_cets_28 / cockroachdb_cets / 0638a02
Reports root/ big
OK: 5315 / Failed: 2 / User-skipped: 195 / Auto-skipped: 0

pubsub_SUITE:dag+collection_config:disable_persist_items_leaf_test

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_disable_persist_items_leaf_test_3687@localhost/res1">>,
          escalus_tcp,<0.115350.0>,
          [{event_manager,<0.115348.0>},
           {server,<<"localhost">>},
           {username,<<"alicE_disable_persist_items_leaf_test_3687">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.115348.0>},
            {server,<<"localhost">>},
            {username,
              <<"alicE_disable_persist_items_leaf_test_3687">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"alice_disable_persist_items_leaf_test_3687">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,fun escalus_auth:auth_plain/2},
           {wspath,undefined},
           {username,<<"alicE_disable_persist_items_leaf_test_3687">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"8821e3d795e972fe">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {pubsub_tools,receive_response,3,
       [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
        {line,444}]},
     {pubsub_tools,receive_and_check_response,4,
       [{file,"/home/circleci/project/big_tests/tests/pubsub...

Report log

pubsub_SUITE:tree+last_item_cache:send_last_published_item_no_items_test

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_send_last_published_item_no_items_test_3749@localhost/res1">>,
          escalus_tcp,<0.116807.0>,
          [{event_manager,<0.116804.0>},
           {server,<<"localhost">>},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3749">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.116804.0>},
            {server,<<"localhost">>},
            {username,
              <<"alicE_send_last_published_item_no_items_test_3749">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_send_last_published_item_no_items_test_3749">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,fun escalus_auth:auth_plain/2},
           {wspath,undefined},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3749">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"320a9fcc46b4e29b">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {pubsub_tools,receive_response,3,
       [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
        {line,444}]},
     {pubsub_tools,receive_and_c...

Report log

mssql_mnesia_28 / odbc_mssql_mnesia / 0638a02
Reports root/ big
OK: 5609 / Failed: 0 / User-skipped: 140 / Auto-skipped: 0

pgsql_mnesia_28 / pgsql_mnesia / 0638a02
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0

ldap_mnesia_27 / ldap_mnesia / 0638a02
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1359 / Auto-skipped: 0

chrzaszcz · 2025-12-05T09:19:49Z

Would it be possible for you to address these few items?

Update the description. It looks like you enable it for big and small tests, but it was already enabled for big tests.
Provide information why the scripts for big tests were also changed. I couldn't find any information about the reasons for these changes.
Hide all the automatic comments with test results as outdated (except the latest one).

tomaszwojcikowski · 2025-12-05T09:47:24Z

Would it be possible for you to address these few items?

Update the description. It looks like you enable it for big and small tests, but it was already enabled for big tests.

Provide information why the scripts for big tests were also changed. I couldn't find any information about the reasons for these changes.

Hide all the automatic comments with test results as outdated (except the latest one).

Changes to big tests: The existing maybe_select_suites() function was refactored to use the new select_circleci_suites() helper function and improved to handle edge cases better
The copy of JUnit report for big tests was changed from using a glob pattern (cp big_tests/ct_report/*/junit_report.xml .) to explicitly finding the latest CT run directory - this fixes a bug where the glob could match multiple directories (from multiple runs) or fail (when there is o such directory). The new implementation works like this:
- It uses ls -1 -t to list directories sorted by time (newest first)
- It pipes through grep ct_run to filter only CT run directories
- It uses head -n1 to guarantee exactly one result (the most recent)

chrzaszcz

Looks good. I added one comment.

.circleci/template.yml

mongoose-im · 2025-12-08T09:23:56Z

elasticsearch_and_cassandra_28 / elasticsearch_and_cassandra_mnesia / 30f28e5
Reports root/ big
OK: 683 / Failed: 0 / User-skipped: 72 / Auto-skipped: 0

small_tests_28 / small_tests / 30f28e5
Reports root / small

small_tests_27 / small_tests / 30f28e5
Reports root / small

small_tests_28_arm64 / small_tests / 30f28e5
Reports root / small

ldap_mnesia_27 / ldap_mnesia / 30f28e5
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1359 / Auto-skipped: 0

ldap_mnesia_28 / ldap_mnesia / 30f28e5
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1359 / Auto-skipped: 0

dynamic_domains_mysql_redis_28 / mysql_redis / 30f28e5
Reports root/ big
OK: 5186 / Failed: 0 / User-skipped: 157 / Auto-skipped: 0

internal_mnesia_28 / internal_mnesia / 30f28e5
Reports root/ big
OK: 2499 / Failed: 0 / User-skipped: 1218 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_28 / pgsql_mnesia / 30f28e5
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

pgsql_cets_28 / pgsql_cets / 30f28e5
Reports root/ big
OK: 5311 / Failed: 0 / User-skipped: 195 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_28 / odbc_mssql_mnesia / 30f28e5
Reports root/ big
OK: 5216 / Failed: 0 / User-skipped: 127 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 30f28e5
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

mysql_redis_28 / mysql_redis / 30f28e5
Reports root/ big
OK: 5600 / Failed: 0 / User-skipped: 149 / Auto-skipped: 0

pgsql_mnesia_27 / pgsql_mnesia / 30f28e5
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0

pgsql_mnesia_28 / pgsql_mnesia / 30f28e5
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0

mssql_mnesia_28 / odbc_mssql_mnesia / 30f28e5
Reports root/ big
OK: 5609 / Failed: 0 / User-skipped: 140 / Auto-skipped: 0

Enable rerun failed tests for small and big tests

1c3de3d

tomaszwojcikowski force-pushed the tomasz/MIM-2364-rerun-flaky-tests branch from 631e9c9 to 1c3de3d Compare December 2, 2025 15:11

tomaszwojcikowski changed the title ~~Add reporting for small test results~~ Enable "Rerun Failed Tests" for Small and Big Tests Dec 2, 2025

tomaszwojcikowski changed the title ~~Enable "Rerun Failed Tests" for Small and Big Tests~~ [CI] Enable "Rerun Failed Tests" for Small and Big Tests Dec 2, 2025

tomaszwojcikowski added the tests label Dec 2, 2025

tomaszwojcikowski requested a review from chrzaszcz December 2, 2025 16:34

Tomasz Wojcikowski added 3 commits December 3, 2025 10:18

Merge branch 'master' into tomasz/MIM-2364-rerun-flaky-tests

bd671bf

Update CircleCI config to always run coverage step and fix Makefile t…

bb3b809

…o include extra args for test suite

Refactor small suite path function to improve log file retrieval

a560aaf

tomaszwojcikowski force-pushed the tomasz/MIM-2364-rerun-flaky-tests branch from 0638a02 to a560aaf Compare December 3, 2025 15:00

esl deleted a comment from mongoose-im Dec 5, 2025

tomaszwojcikowski changed the title ~~[CI] Enable "Rerun Failed Tests" for Small and Big Tests~~ [CI] Enable "Rerun Failed Tests" for Small Tests Dec 5, 2025

chrzaszcz approved these changes Dec 5, 2025

View reviewed changes

.circleci/template.yml Show resolved Hide resolved

Store JUnit report in dedicated directory for big tests

30f28e5

chrzaszcz merged commit b05aeae into master Dec 8, 2025
4 checks passed

chrzaszcz deleted the tomasz/MIM-2364-rerun-flaky-tests branch December 8, 2025 12:39

pawlooss1 added this to the 6.5.0 milestone Dec 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI] Enable "Rerun Failed Tests" for Small Tests #4586

[CI] Enable "Rerun Failed Tests" for Small Tests #4586

tomaszwojcikowski commented Dec 2, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 2, 2025 •

edited

Loading

Uh oh!

mongoose-im commented Dec 3, 2025 •

edited

Loading

Uh oh!

mongoose-im commented Dec 3, 2025 •

edited

Loading

Uh oh!

chrzaszcz commented Dec 5, 2025

Uh oh!

tomaszwojcikowski commented Dec 5, 2025

Uh oh!

chrzaszcz left a comment

Uh oh!

Uh oh!

mongoose-im commented Dec 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[CI] Enable "Rerun Failed Tests" for Small Tests #4586

[CI] Enable "Rerun Failed Tests" for Small Tests #4586

Conversation

tomaszwojcikowski commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mongoose-im commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mongoose-im commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrzaszcz commented Dec 5, 2025

Uh oh!

tomaszwojcikowski commented Dec 5, 2025

Uh oh!

chrzaszcz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mongoose-im commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tomaszwojcikowski commented Dec 2, 2025 •

edited

Loading

codecov bot commented Dec 2, 2025 •

edited

Loading

mongoose-im commented Dec 3, 2025 •

edited

Loading

mongoose-im commented Dec 3, 2025 •

edited

Loading

mongoose-im commented Dec 8, 2025 •

edited

Loading